[Demo][WIP] Storage mounting - #527

👀

Wow, this looks very cool! I will try it out tomorrow.

I was trying to install goofys on Mac OS 11.5, but failed. Even if I install the macfuse first with brew install --cask macfuse restart the computer, and then brew install goofys. It still tells me Error: goofys has been disabled because it requires closed-source macFUSE! I may need to install it from the source, but that would be a hard experience for the user.

I just tried the distirbuted-resnet training with the mounted bucket for checkpoints. It worked well. I used the following yaml file and commands.

1. sky launch -c mount-test task.yaml
2. sky cancel mount-test 1 after 90 epochs
3. sky exec mount-test task.yaml it resumed the training from 90 epochs.

PS, I had to manually comment out the following lines to avoid the checkpoints being overwritten.
https://github.com/sky-proj/sky/blob/e63957ab0529442e7b7d43fbb2c40925b4e31e7a/sky/data/storage.py#L356-L362

name: resnet-distributed-app


resources:
    accelerators: V100

num_nodes: 2

file_mounts:
    /checkpoints:
        name: sky-checkpoints-zhwu
        source: ~/tmp
        mode: MOUNT

setup: |
    pip3 install --upgrade pip
    rm -r ./pytorch-distributed-resnet
    git clone https://github.com/Michaelvll/pytorch-distributed-resnet.git
    cd pytorch-distributed-resnet && pip3 install -r requirements.txt
    mkdir -p data  && mkdir -p saved_models && cd data && \
    wget -c --quiet https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
    tar -xvzf cifar-10-python.tar.gz
    mkdir -p /checkpoints/torch_ddp_resnet/

run: |
    cd pytorch-distributed-resnet
    git pull

    num_nodes=`echo "$SKY_NODE_IPS" | wc -l`
    master_addr=`echo "$SKY_NODE_IPS" | head -n1`
    python3 -m torch.distributed.launch --nproc_per_node=1 \
    --nnodes=$num_nodes --node_rank=${SKY_NODE_RANK} --master_addr=$master_addr \
    --master_port=8008 resnet_ddp.py --num_epochs 100 --model_dir /checkpoints/torch_ddp_resnet/ \
    --resume --model_filename resnet_distributed-with-epochs.pth

Out of curiosity, is it possible to mount the storage under the workdir? Since many ML codebase will save the checkpoint in the same directory of the code, can we mount the storage under ~/sky_workdir/checkpoints? If not, one possible way to solve it is to add the target path of the file_mounts to the exclude list of rsync.

Thanks for the comments everyone! Closing this PR for now.